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Personality  Tests 


•  Modest  scale  validity  &  minor  adverse  impact 

•  Scale  scores  often  computed  as  the  mean  item  rating 
with  some  item  ratings  corrected  for  directionality. 

-  Pursue  physical  activities:  non-reversed  (straight)  coded 

-  Avoid  Physical  Activities:  reversed  coded 

•  Research  Goal  —  Explore  use  of  Profile  Similarity 
Metrics  (PSMs)  and  Consensual  Standards  to  increase 
scale  validity 


Personality  Tests  &  Distance  Metrics 


•  Conventional  and  distance 

scores 

are  redundant:  r 

=  -1.00 

Fitness  Motivation 

Rating  Conventional  Key 

Score 

Distance 

Score 

Sum 

Non-reversed  Items: 

1 

l 

5 

4 

5 

“Like  to  exercise...” 

2 

2 

5 

3 

5 

3 

3 

5 

2 

5 

4 

4 

5 

1 

5 

5 

5 

5 

0 

5 

Reversed  Item  Scores 

1 

5 

1 

0 

5 

“Like  to  watch  TV...” 

2 

4 

1 

1 

5 

3 

3 

1 

2 

5 

4 

2 

1 

3 

5 

5 

1 

1 

4 

5 

^  Suggests  PSMs  may  increase  scale  validity 
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(yjfry  D2  and  Shape 

r\ 

•  Conventional  D  formula: 


D2  =  ^(X-Kj)2/ n  for  item  i  =  1  to  n 


(Eql) 


where  XA  and  Kj  correspond  to  observed  ratings / values  from 
response  profile,  X,  and  the  scoring  key,  K,  for  item  i 

•  Statistical  substitutions  for  sdx2,  sdk2,  and  rxk,  provide: 

D2  =  A2t,i_;_  +  ((n-l)(sdx2  +  sdk2  -  2sdxsdkrxk))/n 

tion  ^mean  £ 

PSMs: 


(Eq  2) 


Elevation 
where  A*,, 


lean 


-  Shape  =  rxk 

-  Delta  =  AElevation2  =  (Xmean  -  KnlCi,n)2 

-  Scatter  =  sdx2 

-  Elevation  =  Xmean 


Keyed 

f-  Descriptive 


Research  Design 


•  Analyze  personality  data  used  to  award  ROTC 
scholarships  (i.e.,  two  years  of  operational  data) 


—  Cadet  Background  Experience  Form  (CBEF):  91  items 

—  Sample  size  by  scale  ranged  from  783  to  1786  because  scale  content 
changed  over  years 


•  Longitudinal  Design:  Validated  measures  against 
2nd-year  program  disenrollment  (reversed  continuance) 

•  Design  Constraints/Threats 

—  Scales  were  not  developed  for  PSM  analyses 

—  Some  scales  do  not  have  reversed  items 

—  Not  all  scales  predict  disenrollment 

—  Disenrollment  is  a  low  probability  event  (p  =  .16) 

—  Validated  personality  data  from  high  school  students  (grades  11 
&  12)  against  program  disenrollment  (university  year  2) 


Expectations  &  Hypotheses 


•  PSMs  account  for  most  of  the  variance  in 
Conventional/ distance  scores:  R2  >  .90 


•  PSMs  computed  with  the  conventional  key  will 
increment  the  prediction  of  distance  scores. 


•  Consensual  scoring  standards  may  enhance  scale 
validity 
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CBEF  Scales 


Ratio:  “Reversed”  to  All  items  Scale 

Definition 

2/4 

Igoai 

Goal  Orientation 

3/8 

PWP 

Past  Withdrawal  Propensity 

3/8 

FM 

Fitness  Motivation 

2/6 

IntLd 

Interest  In  Leadership 

2/7 

Lie 

Social  Desirable  Responding 

2/8 

ES 

Equity  Sensitivity 

1/5 

TFI 

Tolerance  for  Injury 

1/8 

Manip 

Manipulativeness 

1/11 

ST 

Stress  Tolerance 

1/14 

AI 

Army  Identification 

0/6 

SE 

Self  Efficacy 

0/6 

PrLd 

Peer  Leadership 

0/9 

Ach 

Achievement 

0/4 

Cch 

Coaching 

0/4 

Hst 

Hostility  to  Authority 
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Distance  Scores  Regressed  on 

PSMs 

PSMs  account  for  nearly  all  distance  score  variance 

•  Primarily  shape  and  scatter  when  item  ratios  >  .25 

•  Primarily  elevation  when  item  ratios  <  .25 


Scale 

(Item  Ratio) 

R 

[3-Shape 

[3-Scatter 

[3-Delta2 

IGOAL  (2:4) 

.94 

-.94 

.07 

.05 

PWP 

(3:8) 

.99 

-.44 

.70 

.19 

FM 

(3:8) 

.99 

-.48 

-.59 

.13 

IntLd 

(2:6) 

.98 

-.34 

-.73 

.10 

LieCon 

(2:7) 

.91 

.02 

.90 

.05 

ES 

(2:8) 

.99 

-.45 

.40 

.41 

TFfI 

(1:5) 

.96 

-.52 

-.28 

.44 

Manip 

(1:8) 

.99 

-.30 

.20 

.70 

ST 

(1:11) 

.98 

-.32 

-.14 

.75 

AI 

(1:14) 

.97 

-.40 

-.08 

.62 

Model  Statistics:  (df  =  3,  776-1782),  (All  F-statistics  >  2361.733),  all  models  and  coefficients 
significant  unless  otherwise  indicated. 

Lie  Conventional  is  scored  dichotomously. 
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Incremental  Validity  of  PSMs  Over 
Distance  Scores  on  Disenrollment 


•  Documented  potential  validity  gains: 

•  3  of  4  conceptually  relevant  scales 

•  4  of  10  scales  with  reverse  items 


Scale  Hierarchical  Analysis  PSM  PSM  Model  Coefficients 

Model 


R 

Dist  Step 

1 

R 

PSMs 
Step  2 

R2 

change 

Sig 

F  change 

R 

Sig 

F  change 

Shape 

Scatter 

Delta2 

p 

P 

p 

P 

p 

P 

IntLd 

.01 

.14 

.020 

.001 

.14 

.001 

i 

o 

.002 

.124 

.003 

.073 

.060 

AI 

.06 

.12 

.011 

.001 

.09 

.002 

-.021 

ns 

.032 

ns 

.078 

.026 

Manip 

.04 

.11 

.011 

.014 

.09 

.038 

-.009 

ns 

.081 

.015 

-.077 

.104 

Lie 

.07 

.09 

.005 

.061 

.09 

.003 

.052 

.074 

.066 

.005 

-.016 

ns 

No  significant  gains  for  FM,  Igoal,  PWP,  ST,  TFI,  &  ES 


Understanding  Gains 


•  Distance  scores  represent  poorly  weighted  composites  to 
predict  continuance 

•  Compare  regression  equations  for  Distance  versus  Disenrollment 
on  PSMs 


Scale 

Outcome 

PSM  Model  Coefficients 

R 

/2-Shape 

/2-Scatter 

/2-Delta2 

IntLd 

Disenroll 

.15 

-.13 

.12 

.07 

Distance 

.98 

-.34 

-.73 

.10 

AI 

Disenroll 

.13 

-.02 

.03 

.08 

Distance 

.97 

-.40 

-.08 

.62 

Manip 

Disenroll 

.12 

-.01 

.08 

-.08 

Distance 

.99 

-.30 

.20 

.70 

Lie 

Disenroll 

.10 

.05 

.07 

-.02 

Conventional 

.91 

.02 

.90 

.05 
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Incremental  Validity  of  Consensus 
Distance  Over  Simple  Distance 


Scale  Hierarchical  Analysis  Distance  Loadings 

R  R  R2change  Sig  F  change  Simple  CBA 

Simple  D  Consensus  D  j3  p  (2  p 

Step  1  Step  2 


PrLd 

.02 

.14 

.019 

.001 

-.04 

ns 

.139 

.001 

IntLd 

.02 

.15 

.022 

.001 

-.01 

ns 

.151 

.001 

AI 

.06 

.11 

.008 

.001 

.02 

ns 

.101 

.001 

Ach 

.02 

.05 

.002 

.056 

-.01 

ns 

.05 

.056 

•  Suggests  potential  validity  gains  for  the  Peer  Leadership  Scale 

•  Consensus  validity  gains  for  the  Interest  in  Leadership  and  Army 
Identification  do  not  increment  the  PSM  models  for  those  scales 
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Conclusions  /  Questions 


•  Modest  validity  gains  using  PSMs  and  consensual 
standards  to  score  conventional  personality  scales 


—  Distance  scores  represent  poorly  weighted  PSM 
composites 

—  Design  weaknesses  may  minimize  gains 
•  Larger  gains  for  conceptually  relevant  scales: 


—  Interest  in  Leadership 

—  Peer  Lead 

—  Army  Identification 

—  Lie 
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